Image processing method and apparatus, image device, and storage medium

ABSTRACT

Embodiments of the present disclosure disclose an image processing method and apparatus, an image device, and a storage medium. The image processing method includes: obtaining an image; obtaining a feature of a part of a target based on the image; determining movement information of the part based on the feature; and controlling movement of a corresponding part in a controlled model according to the movement information.

CROSS REFERENCE TO RELATED APPLICATION

The present application is a continuation of International ApplicationNo. PCT/CN2020/072526, filed on Jan. 16, 2020, which claims priority toChinese Patent Application No. 201910049830.6, filed on Jan. 18, 2019and entitled “IMAGE PROCESSING METHOD AND APPARATUS, IMAGE DEVICE, ANDSTORAGE MEDIUM”, and Chinese Patent Application No. 201910362107.3,filed on Apr. 30, 2019 and entitled “IMAGE PROCESSING METHOD ANDAPPARATUS, IMAGE DEVICE, AND STORAGE MEDIUM”, all of which areincorporated herein by reference in their entirety.

TECHNICAL FIELD

The present disclosure relates to the field of information technologies,and in particular, to an image processing method and apparatus, an imagedevice, and a storage medium.

BACKGROUND

With the development of information technologies, users perform onlineteaching, webcasting, motion sensing games, etc. by means of videorecording. However, in some cases, for example, motion sensing gamesrequire users to wear special motion sensing devices to detect theactivities of their own body, etc. so as to control game characters.However, during online teaching or webcasting, the user's face or bodyare completely exposed to a network. This may involve a user's privacyissue, and may also involve an information security issue. In order tosolve the privacy or security issue, a face image may be covered bymosaics and the like, but this may affect the video effect.

SUMMARY

In this regard, embodiments of the present disclosure provide an imageprocessing method and apparatus, an image device, and a storage medium.

In a first aspect, the present disclosure provides an image processingmethod, including:

obtaining an image; obtaining a feature of a part of a target based onthe image; determining movement information of the part based on thefeature; and controlling the movement of a corresponding part in acontrolled model according to the movement information.

Based on the foregoing solution, obtaining the feature of the part ofthe target based on the image includes: obtaining a first-type featureof a first-type part of the target based on the image; and/or obtaininga second-type feature of a second-type part of the target based on theimage.

Based on the foregoing solution, obtaining the first-type feature of thefirst-type part of the target based on the image includes: obtaining anexpression feature of a head and an intensity coefficient of theexpression feature based on the image.

Based on the foregoing solution, obtaining the intensity coefficient ofthe expression feature based on the image includes: obtaining, based onthe image, an intensity coefficient that represents each sub-part in thefirst-type part.

Based on the foregoing solution, determining the movement information ofthe part based on the feature includes: determining the movementinformation of the head based on the expression feature and theintensity coefficient; and controlling the movement of the correspondingpart in the controlled model according to the movement informationincludes: controlling an expression change of a head in the controlledmodel according to the movement information of the head.

Based on the foregoing solution, obtaining the second-type feature ofthe second-type part of the target based on the image includes:obtaining position information of a key point of the second-type part ofthe target based on the image; and determining the movement informationof the part based on the feature includes: determining movementinformation of the second-type part based on the position information.

Based on the foregoing solution, obtaining the position information ofthe key point of the second-type part of the target based on the imageincludes: obtaining a first coordinate of a support key point of thesecond-type part of the target based on the image; and obtaining asecond coordinate based on the first coordinate.

Based on the foregoing solution, obtaining the first coordinate of thesupport key point of the second-type part of the target based on theimage includes: obtaining a first 2-Dimensional (2D) coordinate of thesupport key point of the second-type part based on a 2D image; andobtaining the second coordinate based on the first coordinate includes:obtaining a first 3-Dimensional (3D) coordinate corresponding to thefirst 2D coordinate based on the first 2D coordinate and a conversionrelationship between a 2D coordinate and a 3D coordinate.

Based on the foregoing solution, obtaining the first coordinate of thesupport key point of the second-type part of the target based on theimage includes: obtaining a second 3D coordinate of the support keypoint of the second-type part of the target based on a 3D image; andobtaining the second coordinate based on the first coordinate includes:obtaining a third 3D coordinate based on the second 3D coordinate.

Based on the foregoing solution, obtaining the third 3D coordinate basedon the second 3D coordinate includes: correcting, based on the second 3Dcoordinate, a 3D coordinate of a support key point corresponding to anoccluded portion of the second-type part in the 3D image, so as toobtain the third 3D coordinate.

1161Based on the foregoing solution, determining the movementinformation of the second-type part based on the position informationincludes: determining a quaternion of the second-type part based on theposition information.

Based on the foregoing solution, obtaining the position information ofthe key point of the second-type part of the target based on the imageincludes: obtaining first position information of the support key pointof a first part in the second-type part; and obtaining second positioninformation of the support key point of a second part in the second-typepart.

Based on the foregoing solution, determining the movement information ofthe second-type part based on the position information includes:determining movement information of the first part according to thefirst position information; and determining movement information of thesecond part according to the second position information.

Based on the foregoing solution, controlling the movement of thecorresponding part in the controlled model according to the movementinformation includes: controlling movement of a part in the controlledmodel corresponding to the first part according to the movementinformation of the first part; and controlling movement of a part in thecontrolled model corresponding to the second part according to themovement information of the second part.

Based on the foregoing solution, the first part is a torso; and/or thesecond part is upper limbs, lower limbs, or four limbs.

In a second aspect, the present disclosure provides an image processingapparatus, including:

a first obtaining module, configured to obtain an image; a secondobtaining module, configured to obtain a feature of a part of a targetbased on the image; a first determining module, configured to determinemovement information of the part based on the feature; and a controlmodule, configured to control the movement of a corresponding part in acontrolled model according to the movement information.

In a third aspect, the present disclosure provides an image device,including: a memory; and a processor, connected to the memory andconfigured to execute computer-executable instructions stored on thememory so as to implement the image processing method according to anyone of the foregoing items.

In a fourth aspect, the present disclosure provides a non-volatilecomputer storage medium, having computer-executable instructions storedthereon, where after the computer-executable instructions are executedby a processor, the image processing method according to any one of theforegoing items is implemented.

According to the technical solutions provided by the embodiments of thepresent disclosure, the feature of the part of the target is obtainedaccording to the obtained image, then movement information of the partis obtained based on the feature of the part, and finally the movementof the corresponding part in the controlled model is controlledaccording to the movement information. In this way, when the controlledmodel is used to simulate the movement of the target for live videostreaming, the movement of the controlled model may be preciselycontrolled, so that the controlled model precisely simulates themovement of the target. On the one hand, live video streaming isachieved, and on the other hand, user privacy is protected.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a schematic flowchart of an image processing method providedby embodiments of the present disclosure.

FIG. 2 is a schematic flowchart of an image processing method providedby other embodiments of the present disclosure.

FIGS. 3A to 3C are schematic diagrams of simulating a change in acaptured user's hand movement by a controlled model provided by theembodiments.

FIGS. 4A to 4C are schematic diagrams of simulating a change in acaptured user's torso movement by a controlled model provided by theembodiments.

FIGS. 5A to 5C are schematic diagrams of simulating a captured user'sfoot movement by a controlled model provided by the embodiments.

FIG. 6 is a schematic structural diagram of an image processingapparatus provided by embodiments of the present disclosure.

FIG. 7A is a schematic diagram of skeleton key points provided byembodiments of the present disclosure.

FIG. 7B is a schematic diagram of skeleton key points provided by otherembodiments of the present disclosure.

FIG. 8 is a schematic diagram of a skeleton provided by embodiments ofthe present disclosure.

FIG. 9 is a schematic diagram of local coordinate systems of differentbones of a human body provided by embodiments of the present disclosure.

FIG. 10 is a schematic structural diagram of an image device provided byembodiments of the present disclosure.

DETAILED DESCRIPTION

The technical solutions of the present disclosure are further describedin detail below with reference to the accompanying drawings and specificembodiments.

As shown in FIG. 1, the embodiments provide an image processing method,including the following steps.

At step S110, an image is obtained.

At step S120, a feature of a part of a target is obtained based on theimage

At step S130, movement information of the part is determined based onthe feature.

At step S140, the movement of a corresponding part in a controlled modelis controlled according to the movement information.

According to the image processing method provided by the embodiments,the movement of the controlled model may be driven by means of imageprocessing.

The image processing method provided by the embodiments may be appliedto an image device, where the image device may be various electronicdevices that are capable of performing image device processing, such asan electronic device that performs image capture, image display, andimage pixel recombination to generate an image. The image deviceincludes, but is not limited to, various terminal devices, such as amobile device and/or a fixed terminal, and further includes variousimage servers that are capable of providing an image service. The mobileterminal includes a portable device such as a mobile phone or a tabletcomputer a user may easily carry, and further includes a device worn bythe user, such as a smart bracelet, a smart watch, or smart glasses. Thefixed terminal includes a fixed desktop computer, etc.

In the embodiments, the image obtained in step S110 may be atwo-dimensional (2D) or a three-dimensional (3D) image. The 2D image mayinclude an image captured by a monocular or multi-ocular camera, such asa red-green-blue (RGB) image.

The 3D image may be a 3D image obtained based on 2D coordinates detectedfrom a 2D image, and then using a conversion algorithm of the 2Dcoordinates to 3D coordinates. The 3D image may further be an imagecaptured by a 3D camera.

An approach for obtaining the image may include: capturing the imageusing a camera of the image device; and/or, receiving the image from anexternal device; and/or, reading the image from a local database or alocal memory.

In one example, step S120 includes: detecting the image to obtain afeature of one part of a target, where the part is any part on thetarget.

In another example, step S120 includes: detecting the image to obtainfeatures of at least two parts of the target, where the two parts aredifferent parts on the target. The two parts are continuouslydistributed on the target or are distributed on the target at aninterval.

For example, if the target is a person, the any part includes any one ofthe following parts: head, torso, four limbs, upper limbs, lower limbs,hands, feet, etc. The at least two parts include at least two of thefollowing parts: head, torso, four limbs, upper limbs, lower limbs,hands, feet, etc. In some other embodiments, the target is not limitedto a human, but may also be various movable living bodies or non-livingbodies such as animals

In the embodiments, a feature of one or more parts is obtained, wherethe feature may be a feature that represents spatial structureinformation, position information, or movement status of the part invarious forms. In the embodiments, the image may be detected using adeep learning model such as a neural network so as to obtain thefeature.

In one example, the feature represents a relative positionalrelationship between joint points in a human skeleton. In anotherexample, the feature represents a positional change relationship ofcorresponding joint points in a human skeleton at adjacent time points,or the feature represents a positional change relationship ofcorresponding joint points in a human skeleton of a current picture andan initial coordinate system (also referred to as a camera coordinatesystem). More specifically, the feature includes 3D coordinates of jointpoints in a human skeleton detected by the deep learning model (such asa neural network used in an OpenPose project) in a world coordinatesystem. In still another example, the feature includes an optical flowfeature that represents a change in a human posture, etc.

In step S110, the obtained image may be a frame of image or multipleframes of images. For example, when the obtained image is a frame ofimage, the subsequently obtained movement information reflects themovement of a joint point in a current image with respect to acorresponding joint point in the camera coordinate system. For anotherexample, when multiple frames of images are obtained, the subsequentlyobtained movement information reflects the movement of a joint point inthe current image with respect to a corresponding joint point inprevious several frames of images, or the subsequently obtained movementinformation also reflects the movement of a joint point in the currentimage with respect to a corresponding joint point in the cameracoordinate system. The number of obtained images is not limited in thepresent application.

After the feature is obtained, movement information of the part isobtained, where the movement information represents an action change ofthe corresponding part and/or an expression change caused by the actionchange, etc.

In one example, assuming that the two parts involved in S120 are thehead and the torso, in step S140, a part in the controlled modelcorresponding to the head is controlled to move, and a part in thecontrolled model corresponding to the torso is controlled to move.

The movement information includes, but is not limited to, coordinates ofa key point corresponding to the part, where the coordinates include,but are not limited to, 2D coordinates and 3D coordinates. Thecoordinates represent a change in the key point corresponding to thepart with respect to a reference position, so as to represent themovement status of the corresponding part. The movement information maybe expressed in various information forms such as a vector, an array, aone-dimensional value, and a matrix.

The controlled model may be a model corresponding to the target. Forexample, if the target is a person, the controlled model is a humanmodel; if the target is an animal, the controlled model is a body modelof the corresponding animal; and if the target is a transportation tool,the controlled model is a model of the transportation tool.

In the embodiments, the controlled model is a model for the category towhich the target belongs. The model may be predetermined and may befurther divided into multiple styles. The style of the controlled modelmay be determined based on a user instruction, and the controlled modelmay include multiple styles, such as a real-person style that simulatesa real person, a comics and animation style, a network celebrity style,styles with different temperaments, and a game style. The styles withdifferent temperaments may be literary style or a rock style. In thegame style, the controlled model may be a character in the game.

For example, in the process of online teaching, some teachers are notwilling to expose their face and body, thinking that this is privacy. Ifa video is directly recorded, the teacher's face and body are inevitablyexposed. In the embodiments, an image of the movement of the teacher maybe obtained by means of image capture, etc., and then a virtualcontrolled model movement is controlled by means of feature extractionand obtaining of movement information. In this way, on the one hand, thecontrolled model may simulate the movement of the teacher to completebody movement teaching through its own body movement, and on the otherhand, the movement of the controlled model is used for teaching, theteacher's face and body are not directly exposed to the teaching video,and thus the privacy of the teacher is protected.

For another example, in a road surface surveillance video, if a video ofa vehicle is directly captured, once the video is exposed to thenetwork, all vehicle information of some specific users are exposed, butif surveillance is not performed, there may be a case where theresponsibility cannot be determined when a traffic accident occurs. Ifthe method according to the embodiments is used, a real vehicle movementis simulated with a vehicle model to obtain a surveillance video,license plate information of a vehicle and/or the overall outer contourof the vehicle is retained in the surveillance video, and the brand,model, color, ageing condition, etc. of the vehicle may be hidden,thereby protecting user privacy.

In some embodiments, as shown in FIG. 2, step S120 includes thefollowing steps.

At step S121, a first-type feature of a first-type part of the target isobtained based on the image.

At step S122, a second-type feature of a second-type part of the targetis obtained based on the image.

In the embodiments, the first-type feature and the second-type featureare features that represent spatial structure information, positioninformation, and/or movement status of the corresponding part.

Different types of features have different characteristics, and applyingto different types of parts may have higher precision. For example, interms of a muscle movement of a human face with respect to the movementof four limbs, the precision of a spatial change caused by differentfeatures to the movement is different. In this case, in the embodiments,for the face and the four limbs, different types of features with theprecision respectively adapted to the human face or the four limbs maybe used for representation.

In some embodiments, for example, a first-type feature of a first-typepart and a second-type feature of a second-type part are respectivelyobtained based on the image.

The first-type part and the second-type part are different types ofparts; and different types of parts may be distinguished by theamplitudes of movement of different types of parts or distinguishedusing movement fineness of different types of parts.

In the embodiments, the first-type part and the second-type part may betwo types of parts with a relatively large difference in the maximumamplitude of movement. The first-type part may be the head. The fivesense organs of the head may move, but the amplitudes of movement of thefive sense organs of the head are relatively small. The whole head mayalso move, for example, nodding or shaking, but the amplitude ofmovement is relatively small compared to the amplitude of movement ofthe limb or torso.

The second-type part may be upper limbs, lower limbs, or four limbs, andthe amplitude of the limb movement is very large. If the movementstatuses of the two types of parts are represented by the same feature,it may cause problems such as a decrease in precision or an increase inthe complexity of an algorithm because of the amplitude of movement of acertain part.

Herein, according to the characteristics of different types of parts,the movement information is obtained with different types of features.Compared with the related approach of using the same type of features torepresent the same type of parts, the precision of information of atleast one type of part may be increased and the precision of themovement information may be improved.

In some embodiments, the obtaining subjects of the first-type featureand the second-type feature are different, for example, using differentdeep learning models or deep learning modules. The obtaining logic ofthe first-type feature is different from that of the second-typefeature.

In some embodiments, step S121 includes: obtaining an expression featureof a head based on the image.

In the embodiments, the first-type part is a head, the head includes aface, and the expression feature includes, but is not limited to, atleast one of the following: eyebrow movement, mouth movement, nosemovement, eye movement, or cheek movement. The eyebrow movement mayinclude: raising eyebrows and drooping eyebrows. The mouth movement mayinclude: opening the mouth, closing the mouth, twitching the mouth,pouting, grinning, baring teeth, etc. The nose movement may include: thecontraction of the nose caused by inhaling into the nose, andoutward-blowing accompanying nose extension movement. The eye movementmay include, but is not limited to: orbital movement and/or eyeballmovement. The orbital movement may change the size and/or shape of theorbit, for example, the shape and size of the orbit may change duringsquinting, glaring, and smiling of eyes. The eyeball movement mayinclude: the position of the eyeball in the orbit, for example, thechange in the user's line of sight may cause the eyeball to be locatedat different positions of the orbit, the movement of the eyeballs of theleft and right eyes together may reflect different emotional states ofthe user, etc. For the cheek movement, dimples or pear vortexes areproduced when some users smile, and the shape of the cheek also changesaccordingly.

In some embodiments, the head movement is not limited to the expressionmovement, and then the first-type feature is not limited to theexpression feature and also includes: hair movement features of movementof hair of the head, etc.; and the first-type feature may furtherinclude: overall head movement features of shaking the head and/ornodding the head, etc.

In some embodiments, step S121 further includes: obtaining an intensitycoefficient of the expression feature based on the image.

In the embodiments, the intensity coefficient may correspond to theexpression amplitude of a facial expression. For example, multipleexpression bases are set on the face, and one expression basecorresponds to one expression action. The intensity coefficient hereinmay be used for representing the strength of the expression action, forexample, the strength is the amplitude of the expression action.

In some embodiments, the greater the intensity coefficient is, thehigher the strength represented is. For example, the higher theintensity coefficient is, the greater the amplitude of the mouth-openingexpression base is, and the greater the amplitude of the poutingexpression base is. For another example, the greater the intensitycoefficient is, the higher the eyebrow-raising height for theeyebrow-raising expression base is.

By introducing the intensity coefficient, not only the controlled modelmay simulate the current action of the target, but also the strength ofthe current expression of the target may be precisely simulated, so asto achieve precise migration of expression. In this way, if the methodis applied to a motion-sensing game scenario, the controlled object is agame character. By using this method, the game character not only may becontrolled by the body movement of the user, but also may preciselysimulate the expression features of the user. In this way, in the gamescenario, the degree of simulation of the game scenario is increased,and the user's game experience is improved.

In the embodiments, when the target is a person, mesh informationrepresenting the expression change of the head is obtained by means ofmesh detection, etc., and the change in the controlled model iscontrolled based on the mesh information. The mesh information includes,but is not limited to: quadrilateral mesh information and/or trianglepatch information. The quadrilateral mesh information indicatesinformation of longitude and latitude lines, and the triangle patchinformation is information of a triangle patch formed by connectingthree key points.

For example, the mesh information is formed by a predetermined number offace key points including a face body surface, the intersection point ofthe longitude and latitude lines in the mesh represented by thequadrilateral mesh information may be the position of the face keypoints, and the change in the position of the intersection point of themesh is the expression change. In this way, the expression feature andintensity coefficient obtained based on the quadrilateral meshinformation may be used for precisely controlling the expression of theface of the controlled model. For another example, the vertices of atriangle patch corresponding to the triangle patch information includeface key points, and the change in the position of the key points is theexpression change. The expression feature and intensity coefficientobtained based on the triangle patch information may be used for precisecontrol of the facial expression of the controlled model.

In some embodiments, obtaining the intensity coefficient of theexpression feature includes: obtaining, based on the image, an intensitycoefficient that represents each sub-part in the first-type part.

For example, the five sense organs of the face, i.e., eyes, eyebrows,nose, mouth, and ears, respectively correspond to at least oneexpression base, and some correspond to multiple expression bases, andone expression base corresponds to a type of expression action of asense organ, while the intensity coefficient represents the amplitude ofthe expression action.

In some embodiments, step S130 includes: determining movementinformation of the head based on the expression feature and theintensity coefficient; and step S140 includes: controlling an expressionchange of a corresponding head in the controlled model according to themovement information of the head.

In some embodiments, step S122 includes: obtaining position informationof a key point of the second-type part of the target based on the image.

The position information may be represented by position information ofkey points of the target, and the key points include: support key pointsand outer contour key points. If a person is taken as an example, thesupport key points include skeleton key points of a human body, and thecontour key points may be key points of an outer contour of a human bodysurface. The number of key points is not limited in the presentapplication, but the key points represent at least a portion of theskeleton.

The position information may be represented by coordinates, e.g.,represented by 2D coordinates and/or 3D coordinates in the predeterminedcoordinate system. The predetermined coordinate system includes, but isnot limited to, an image coordinate system where an image is located.The position information may be the coordinates of key points, and isobviously different from the foregoing mesh information. Because thesecond-type part is different from the first-type part, the change inthe movement of the second-type part may be more precisely representedby using the position information.

In some embodiments, step S130 includes: determining movementinformation of the second-type part based on the position information.

If the target being a person is taken as an example, the second-typepart includes, but is not limited to: a torso and/or four limbs; and atorso and/or upper limbs, and a torso and/or lower limbs.

Furthermore, step S122 specifically includes: obtaining a firstcoordinate of a support key point of the second-type part of the targetbased on the image; and obtaining a second coordinate based on the firstcoordinate.

The first coordinate and the second coordinate are both coordinates thatrepresent the support key point. If the target being a person or ananimal is taken as an example, the support key point herein is askeleton key point.

The first coordinate and the second coordinate may be different types ofcoordinates. For example, the first coordinate is a 2D coordinate in the2D coordinate system, and the second coordinate is a 3D coordinate inthe 3D coordinate system. The first coordinate and the second coordinatemay also be the same type of coordinates. For example, the secondcoordinate is a coordinate after the first coordinate is corrected, andin this case, the first coordinate and the second coordinate are thesame type of coordinates. For example, the first coordinate and thesecond coordinate are 3D coordinates or 2D coordinates.

In some embodiments, obtaining the first coordinate of the support keypoint of the second-type part of the target based on the image includes:obtaining a first 2D coordinate of the support key point of thesecond-type part based on a 2D image; and obtaining the secondcoordinate based on the first coordinate includes: obtaining a first 3Dcoordinate corresponding to the first 2D coordinate based on the first2D coordinate and a conversion relationship between a 2D coordinate anda 3D coordinate.

In some embodiments, obtaining the first coordinate of the support keypoint of the second-type part of the target based on the image includes:obtaining a second 3D coordinate of the support key point of thesecond-type part of the target based on a 3D image; and obtaining thesecond coordinate based on the first coordinate includes: obtaining athird 3D coordinate based on the second 3D coordinate.

For example, the 3D image directly obtained in step S110 includes: a 2Dimage and a depth image corresponding to the 2D image. The 2D image mayprovide coordinate values of the support key point in a xoy plane, andthe depth value in the depth image may provide coordinates of thesupport key point on a z axis. The z axis is perpendicular to the xoyplane.

In some embodiments, obtaining the third 3D coordinate based on thesecond 3D coordinate includes: correcting, based on the second 3Dcoordinate, a 3D coordinate of a support key point corresponding to anoccluded portion of the second-type part in the 3D image so as to obtainthe third 3D coordinate.

In the embodiments, the second 3D coordinate is first extracted from the3D image using a 3D model, and then the blocking of different parts inthe target is considered. Through correction, correct third 3Dcoordinates of different parts of the target in a 3D space may beobtained, thereby ensuring subsequent control precision of thecontrolled model.

In some embodiments, step S130 includes: determining a quaternion of thesecond-type part based on the position information.

For a specific method for determining the quaternion based on theposition information, please refer to the subsequent description inExample 3.

In some embodiments, the movement information is not only represented bythe quaternion, but also represented by coordinate values in differentcoordinate systems; for example, coordinate values in an Euleriancoordinate system or a Lagrangian coordinate system, etc. By using thequaternion, the spatial position and/or rotation in different directionsof the second-type part may be precisely described.

In some embodiments, the quaternion is taken as the movementinformation. In specific implementation, it is not limited to thequaternion, but may also be indicated by coordinate values in variouscoordinate systems with respect to a reference point, for example, thequaternion may be replaced with the Eulerian coordinates or Lagrangiancoordinates.

In some embodiments, step S120 includes: obtaining first positioninformation of the support key point of a first part in the second-typepart; and obtaining second position information of the support key pointof a second part in the second-type part.

The second-type part may include at least two different parts. Thus, thecontrolled model may simultaneously simulate the movement of at leasttwo parts of the target.

In some embodiments, step S130 includes: determining movementinformation of the first part according to the first positioninformation; and determining movement information of the second partaccording to the second position information.

In some embodiments, step S140 includes: controlling movement of a partin the controlled model corresponding to the first part according to themovement information of the first part; and controlling movement of apart in the controlled model corresponding to the second part accordingto the movement information of the second part.

In some other embodiments, the first part is a torso; and the secondpart is upper limbs, lower limbs, or four limbs.

In some embodiments, the method further includes: determining a secondtype of movement information of a connecting portion according tofeatures of the at least two parts and a first movement constraintcondition of the connecting portion, where the connecting portion isused for connecting the two parts; and controlling movement of theconnecting portion of the controlled model according to the second typeof movement information.

In some embodiments, movement information of some parts is obtainedseparately by means of the movement information obtaining model, and themovement information obtained in this way is referred to as a first typeof movement information. Moreover, some parts are connecting portionsfor connecting other two or more parts, and the movement information ofthese connecting portions is referred to as the second type of movementinformation in the embodiments for convenience. The second type ofmovement information herein is also one of information that representsthe movement status of the part in the target. In some embodiments, thesecond type of movement information is determined based on the firsttype of movement information of the two parts connected by theconnecting portion.

Therefore, the second type of movement information differs from thefirst type of movement information in that: the second type of movementinformation is the movement information of the connecting portion, whilethe first type of movement information is movement information of partsother than the connecting portion; and the first type of movementinformation is generated separately based on the movement status of thecorresponding part, and the second type of movement information may berelated to the movement information of other parts connected to thecorresponding connecting portion.

In some embodiments, step S140 includes: determining a control mode forcontrolling the connecting portion according to the type of theconnecting portion; and controlling the movement of the connectingportion of the controlled model according to the control mode and thesecond type of movement information.

The connecting portion may be used for connecting the other two parts,for example, taking a person as an example, the neck, a wrist, an ankle,and a waist are all connecting portions for connecting the two parts.

The movement information of these connecting portions may beinconvenient to detect or depend on other adjacent parts to a certainextent. Therefore, in the embodiments, the movement information of theconnecting portion may be determined according to the first type ofmovement information of the two or more other parts connected to theconnecting portion, so as to obtain the second type of movementinformation of the corresponding connecting portion.

In the embodiments, considering special information such as an approachfor obtaining the movement information of the connecting portion and theconstraint condition, a corresponding control mode is determinedaccording to the type of the connecting portion, so as to achieveprecise control of the corresponding connecting portion in thecontrolled model.

For example, the lateral rotation of the wrist, for example, therotation by taking the direction in which an upper arm extends to thehand as an axis, is caused by the rotation of the upper arm.

For another example, the lateral rotation of the ankle, for example, therotation by taking the extension direction of the crus as an axis, isalso directly driven by the crus. Certainly, it is also possible thatthe crus is driven by the thigh, and the ankle is further driven by thecrus.

Moreover, for the connecting portion, the neck, its rotation determinesthe orientation of the face and the orientation of the torso.

In some other embodiments, determining the control mode for controllingthe connecting portion according to the type of the connecting portionincludes: if the connecting portion is a first type of connectingportion, determining to use a first type of control mode, where thefirst type of control mode is used for directly controlling the movementof the connecting portion corresponding to the first type of connectingportion in the controlled model.

In some embodiments, the first type of connecting portion is driven byits rotation but not driven by the other parts.

In some other embodiments, the connecting portion further includes asecond type of connecting portion other than the first type ofconnecting portion. The movement of the second type of connectingportion herein may not be limited to itself, but driven by the otherparts.

In some embodiments, determining the control mode for controlling theconnecting portion according to the type of the connecting portionincludes: if the connecting portion is a second type of connectingportion, determining to use a second type of control mode, where thesecond type of control mode is used for indirectly controlling themovement of the second type of connecting portion by controlling theparts other than the second type of connecting portion of the controlledmodel.

The parts other than the second type of connecting portion include, butare not limited to: a part directly connected to the second type ofconnecting portion, or a part indirectly connected to the second type ofconnecting portion.

For example, when the wrist is rotated laterally, it may be that theentire upper limb is moving, and then a shoulder and an elbow arerotating, so that the rotation of the wrist may be indirectly driven bycontrolling the lateral rotation of the shoulder and/or the elbow.

In some embodiments, controlling the movement of the connecting portionof the controlled model according to the control mode and the secondtype of movement information includes: if the control mode is the secondtype of control mode, splitting the second type of movement informationto obtain a first type of rotation information of the connectingportion, the rotation of which is caused by a pull portion; adjustingmovement information of the pull portion according to the first type ofrotation information; and controlling the movement of the pull portionin the controlled model by using the adjusted movement information ofthe pull portion so as to indirectly control the movement of theconnecting portion.

In the embodiments, the first type of rotation information is notrotation information generated by the movement of the second type ofconnecting portion, but movement information of the second type ofconnecting portion generated with respect to a specific reference point(e.g. the center of the human body) of the target when the second typeof connecting portion is pulled by the movement of the other parts(i.e., the pull portion) connected to the second type of connectingportion.

In the embodiments, the pull portion is a part directly connected to thesecond type of connecting portion. Taking the wrist being the secondtype of connecting portion as an example, the pull portion is the elbowabove the wrist or even the shoulder. Taking the ankle being the secondtype of connecting portion is taken as an example, the pull portion isthe knee above the ankle or even the root of the thigh.

The lateral rotation of the wrist along the straight direction from theshoulder to the elbow and to the wrist may be a rotation caused by theshoulder or the elbow, and when the movement information is detected, itis caused by the movement of the wrist. Thus, the lateral rotationinformation of the wrist essentially should be assigned to the elbow orthe shoulder. By means of such transfer assignment, the movementinformation of the elbow or the shoulder is adjusted, and the adjustedmovement information is used to control the movement of the elbow or theshoulder of the controlled model. Thus, the lateral rotationcorresponding to the elbow or the shoulder, as seen from the effect barof the image, is reflected by the wrist of the controlled model, so thatthe movement of the target is precisely simulated by the controlledmodel.

In some embodiments, the method further includes: splitting the secondtype of movement information to obtain a second type of rotationinformation of the rotation of the second type of connecting portionwith respect to the pull portion; and controlling the rotation of theconnecting portion with respect to the pull portion in the controlledmodel by using the second type of rotation information.

The first type of rotation information is information obtained by aninformation model that extracts the rotation information directlyaccording to the features of the image, and the second type of rotationinformation is rotation information obtained by adjusting the first typeof rotation information. In the embodiments, first of all, the movementinformation of the second type of connecting portion with respect to apredetermined posture may be known through the features of the secondtype of connecting portion, for example, the 2D coordinates or the 3Dcoordinates, and the movement information is referred to as the secondtype of movement information. The second type of movement informationincludes, but is not limited to, rotation information.

In some embodiments, the second type of connecting portion includes: awrist and an ankle.

In some other embodiments, if the second type of connecting portion isthe wrist, the pull portion corresponding to the wrist includes: aforearm and/or an upper arm; and/or if the second type of connectingportion is the ankle, the pull portion corresponding to the ankleincludes: a crus and/or a thigh.

In some embodiments, the first type of connecting portion includes aneck connecting the head and the torso.

In still some embodiments, determining the movement information of theconnecting portion according to the features of the at least two partsand the first movement constraint condition of the connecting portionincludes: determining orientation information of the at least two partsaccording to the features of the at least two parts; determiningalternative orientation information of the connecting portion accordingto the orientation information of the at least two parts; anddetermining the movement information of the connecting portion accordingto the alternative orientation information and the first movementconstraint condition.

In some embodiments, determining the alternative orientation informationof the connecting portion according to the orientation information ofthe at least two parts includes: determining a first alternativeorientation and a second alternative orientation of the connectingportion according to the orientation information of the at least twoparts.

Two included angles may be formed between the orientation information ofthe two parts, and the two included angles correspond to the rotationinformation of different orientations of the connecting portion.Therefore, the orientations respectively corresponding to the twoincluded angles are alternative orientations. Only one of the twoalternative orientations satisfies the first movement constraintcondition of the movement of the connecting portion, and therefore, thesecond type of movement information needs to be determined according toa target orientation of the first movement constraint condition. In theembodiments, the included angle of rotation satisfying the firstmovement constraint condition is taken as the second type of movementinformation.

For example, two included angles are formed between the orientation ofthe face and the orientation of the torso, and the sum of the twoincluded angles is 180 degrees. It is assumed that the two includedangles are a first included angle and a second included angle,respectively. Moreover, the first movement constraint condition for theneck connecting the face and the torso is between −90 degrees and 90degrees, and then angles exceeding 90 degrees are excluded according tothe first movement constraint condition. In this way, abnormalities thatthe rotation angle exceeds 90 degrees clockwise or counterclockwise,e.g., 120 degrees and 180 degrees, may be reduced in the process thatthe controlled model simulates the movement of the target. If the firstmovement constraint condition is between −90 degrees and 90 degrees, thefirst movement constraint condition corresponds to two extreme angles.One is −90 degrees and the other is 90 degrees.

However, if the rotation angle exceeds the range of −90 degrees to 90degrees, the detected rotation angle is modified as the maximum angledefined by the first movement constraint condition. For example, if arotation angle exceeding 90 degrees is detected, the detected rotationangle is modified as an extreme angle closer to the detected rotationangle, for example, 90 degrees.

In some embodiments, determining the movement information of theconnecting portion according to the alternative orientation informationand the first movement constraint condition includes: selecting targetorientation information within an orientation change constraint rangefrom the first alternative orientation information and the secondalternative orientation information; and determining the movementinformation of the connecting portion according to the targetorientation information.

For example, taking the neck as an example, the face faces right, andthen the corresponding orientation of the neck may be 90 degreesrightward or 270 degrees leftward. However, according to thephysiological structure of the human body, the orientation of the neckof the human body may not be changed by rotating 270 degrees leftward sothat the neck faces right. In this case, the orientation of the neck,rightward 90 degrees and leftward 270 degrees, are both the alternativeorientation information, the orientation information of the neck needsto be further determined, and needs to be determined according to theforegoing first movement constraint condition. In this example, therightward 90 degrees of the neck is the target orientation informationof the neck, and according to the rightward 90 degrees of the neck, thesecond type of movement information of the neck with respect to thecamera coordinate system is rotating 90 degrees rightward.

The target orientation information herein is information that satisfiesthe first movement constraint condition.

In some embodiments, determining the orientation information of the atleast two parts according to the features of the at least two partsincludes: obtaining a first key point and a second key point of each ofthe at least two parts; obtaining a first reference point of each of theat least two parts, where the first reference point is a predeterminedkey point within the target; generating a first vector based on thefirst key point and the first reference point, and generating a secondvector based on the second key point and the first reference point; anddetermining orientation information of each of the at least two partsbased on the first vector and the second vector.

If the first part in the two parts is the shoulder of the human body,the first reference point of the first part is a waist key point of thetarget or a midpoint of key points of two hips. If the second part inthe two parts is the face, the first reference point of the second partis a connecting point of the neck connected to the face and theshoulder.

The first reference point and the corresponding two key points areconnected to form two vectors, and then the two vectors arecross-multiplied to obtain a normal vector of the two vectors. Thedirection of the normal vector may be regarded as the orientation of thecorresponding part. Therefore, in some embodiments, determining theorientation information of each of the at least two parts based on thetwo vectors includes: cross-multiplying the first vector and the secondvector of one part to obtain the normal vector of a plane where thecorresponding part is located; and taking the normal vector as theorientation information of the part.

If the normal vector is determined, the orientation of the plane wherethe part is located is also determined.

In some embodiments, determining the movement information of theconnecting portion based on the movement information of the at least twoparts includes: obtaining a fourth 3D coordinate of the connectingportion with respect to a second reference point; and obtaining absoluterotation information of the connecting portion according to the fourth3D coordinate; and controlling the movement of the corresponding part inthe controlled model according to the movement information includes:controlling the movement of the corresponding connecting portion of thecontrolled model based on the absolute rotation information.

In some embodiments, the second reference point may be one of thesupport key points of the target, and taking the target being a personas an example, the second reference point may be a key point of theparts connected by the first type of connecting portion. For example,taking the neck as an example, the second reference point may be a keypoint of the shoulder connected to the neck.

In some other embodiments, the second reference point may be the same asthe first reference point, for example, the first reference point andthe second reference point both may be root nodes of the human body, andthe root node of the human body may be a midpoint of a connecting lineof two key points of the hips of the human body. The root node includes,but is not limited to, a key point 0 shown in FIG. 7B. FIG. 7B is aschematic diagram of the skeleton of the human body. In FIG. 7B, a totalof 17 skeleton joint points with labels 0 to 16 are included.

In some other embodiments, controlling the movement of the correspondingconnecting portion of the controlled model based on the absoluterotation information further includes: splitting the absolute rotationinformation according to a pull hierarchical relationship between themultiple connecting portions in the target to obtain relative rotationinformation; and controlling the movement of the correspondingconnecting portion in the controlled model based on the relativerotation information.

For example, the following is an example of one hierarchicalrelationship: the first level: pelvis; the second level: waist; thethird level: thighs (e.g., left thigh and right thigh); the fourthlevel: caves (e.g., left crus and right crus); and the fifth level:feet.

For another example, the following is another hierarchical relationship:the first level: chest; the second level: neck; and the third level:head.

Further, for example, the following is still another hierarchicalrelationship: the first level: clavicles, corresponding to the shoulder;the second level: upper arms; the third level: forearms (also referredto as lower arms); and the fourth level: hand.

From the first level to the fifth level, the hierarchical relationshipdecreases in sequence. The movement of the part at the upper levelaffects the movement of the part at the lower level. Therefore, thelevel of the pull portion is higher than that of the connecting portion.

During determination of the second type of movement information, first,the movement information of the key points corresponding to the part ateach level is obtained, and then based on the hierarchical relationship,the movement information (i.e., the relative rotation information) ofthe key points of the part at the low level with respect to the keypoints of the part at the high level is determined.

For example, if a quaternion is used for representing movementinformation, the relative rotation information may be represented by thefollowing calculation formula (1). A rotation quaternion of each keypoint with respect to a camera coordinate system is {Q₀, Q₁, . . . Q₁₈},and then a rotation quaternion q_(i) of each key point with respect to aparent key point is calculated.

q _(i) =Q _(parent(i)) ⁻¹ _(rent(i)) ·Q _(i)  (1)

where the parent key point parent(i) is a key point at the previouslevel of the current key point i. Q₁ is a rotation quaternion of thecurrent key point i with respect to the camera coordinate system, andQ_(parent(i)) ⁻¹ is an inverse rotation parameter of the key point atthe previous level. For example, Q_(parent(i)) is a rotation parameterof the key point at the previous level, and the rotation angle is 90degrees; and then the rotation angle of Q_(parent(i)) ⁻¹ is −90 degrees.

In some embodiments, controlling the movement of the correspondingconnecting portion of the controlled model based on the absoluterotation information further includes: correcting the relative rotationinformation according to a second constraint condition; and controllingthe movement of the corresponding connecting portion in the controlledmodel based on the relative rotation information includes: controllingthe movement of the corresponding connecting portion in the controlledmodel based on the corrected relative rotation information.

In some embodiments, the second constraint condition includes: arotatable angle of the connecting portion.

In some embodiments, the method further includes: performing posturedefect correction on the second type of movement information to obtaincorrected second type of movement information; and controlling themovement of the connecting portion of the controlled model according tothe second type of movement information includes: controlling themovement of the connecting portion of the controlled model by using thecorrected second type of movement information.

For example, some users have a problem that the shape of the body is notvery standard and a problem of uncoordinated walking, etc. In order toreduce the phenomena that the controlled model directly imitatesrelatively strange movements, etc. In the embodiments, posture defectcorrection may be performed on the second type of movement informationto obtain the corrected second type of movement information.

In some embodiments, the method further includes: performing posturedefect correction on the first type of movement information to obtaincorrected first type of movement information. Step S140 includes:controlling the movement of the corresponding part in the controlledmodel by using the corrected first type of movement information.

In some embodiments, the posture defect correction includes at least oneof the following: an ipsilateral defect of upper and lower limbs; abowleg movement defect; a splayfoot movement defect; or a pigeon-toemovement defect.

In some embodiments, the method further includes: obtaining a posturedefect correction parameter according to difference information betweena body form of the target and a standard body form, where the posturedefect correction parameter is used for correcting the first type ofmovement information or the second type of movement information.

For example, before controlling the controlled model using the imageincluding the target, the body form of the target is detected first, andthen the detected body form is compared with the standard body form toobtain difference information; and posture defect correction isperformed by means of the difference information.

A prompt about maintaining a predetermined posture is output on adisplay interface, and a user maintains the predetermined posture afterseeing the prompt, so that the image device may capture an image of theuser maintaining the predetermined posture. Then, whether thepredetermined posture maintained by the user is standard enough isdetermined by means of image detection to obtain the differenceinformation. The predetermined posture may include, but is not limitedto, an upright posture of the human body.

For example, some persons have splayfeet, while a normal standardstanding posture should be that connecting lines between tiptoes andheels of the feet are parallel. After obtaining the first type ofmovement information and/or the second type of movement informationcorresponding to features of the target, such non-standard correction inthe body form (i.e., the posture defect correction) is considered whenthe controlled model is controlled.

In some other embodiments, the method further includes: correcting aproportion of different parts of a standard model according to aproportional relationship between different parts of the target toobtain a corrected controlled model.

There may be differences in the proportional relationships between partsof different targets. For example, taking a person as an example, theproportion of the leg length to the head length of a professional modelis greater than that of an ordinary person. Some persons have fullbuttocks, and then the distances between their hips may be greater thanthat of an ordinary person.

The standard model may be a mean value model obtained based on a largeamount of human body data. In order to make the controlled model moreprecisely simulate the movement of the target, in the embodiments, theproportions of different parts of the standard model are correctedaccording to the proportional relationship between different parts ofthe target to obtain the corrected controlled model. For example, takingthe target being a person as an example, the corrected parts include,but are not limited to, the hips and/or legs.

As shown in FIGS. 3A, 3B, and 3C, the small image in the upper leftcorner of the image is the captured image, and in the lower right corneris the controlled model of the human body. The user's hand moves. FromFIG. 3A to FIG. 3B and then from FIG. 3B to FIG. 3C, the user's handmoves, and the hand of the controlled model also moves. The user's handmovement sequentially changes from first clenching to palm extension andthen to index finger extension in FIGS. 3A to 3C, while the controlledmodel simulates the user's gesture to change from first clenching topalm extension and then to index finger extension.

As shown in FIGS. 4A, 4B, and 4C, the small image in the upper leftcorner of the image is a captured image, and in the lower right corneris the controlled model of the human body. The user's torso moves. FromFIG. 4A to FIG. 4B and then from FIG. 4B to FIG. 4C, the user's torsomoves, and the torso of the controlled model also moves. From FIGS. 4Ato 4C, the user thrusts the hips toward the right of the image, thruststhe hips toward the left of the image, and finally stands upright. Thecontrolled model also simulates the user's torso movement.

As shown in FIGS. 5A, 5B, and 5C, the small image in the upper leftcorner of the image is a captured image, and in the lower right corneris the controlled model of the human body. From FIGS. 5A to 5C, the usertakes a step toward the right of the image, takes a step toward the leftof the image, and finally stands up straight. The controlled model alsosimulates the user's foot movement.

In addition, in FIGS. 4A to 4C, the controlled model also simulates theuser's expression change.

As shown in FIG. 6, the embodiments provide an image processingapparatus, including the following modules:

a first obtaining module 110, configured to obtain an image;

a second obtaining module 120, configured to obtain a feature of a partof a target based on the image;

a first determining module 130, configured to determine movementinformation of the part based on the feature; and

a control module 140, configured to control the movement of acorresponding part in a controlled model according to the movementinformation.

In some embodiments, the second obtaining module 120 is specificallyconfigured to: obtain a first-type feature of a first-type part of thetarget based on the image; and/or obtain a second-type feature of asecond-type part of the target based on the image.

In some embodiments, the second obtaining module 120 is specificallyconfigured to obtain an expression feature of a head and an intensitycoefficient of the expression feature based on the image.

In some embodiments, obtaining the intensity coefficient of theexpression feature based on the image includes: obtaining, based on theimage, an intensity coefficient that represents each sub-part in thefirst-type part.

In some embodiments, the first determining module 130 is specificallyconfigured to determine the movement information of the head based onthe expression feature and the intensity coefficient; and the controlmodule 140 is specifically configured to control an expression change ofa head in the controlled model according to the movement information ofthe head.

In some embodiments, the second obtaining module 120 is configured toobtain mesh information of the first-type part based on the image.

In some embodiments, the second obtaining module 120 is specificallyconfigured to obtain, based on the image, an intensity coefficient thatrepresents each sub-part in the first-type part.

In some embodiments, the second obtaining module 120 is specificallyconfigured to obtain position information of a key point of thesecond-type part of the target based on the image; and the firstdetermining module 130 is specifically configured to determine movementinformation of the second-type part based on the position information.

In some embodiments, the second obtaining module 120 is specificallyconfigured to: obtain a first coordinate of a support key point of thesecond-type part of the target based on the image; and obtain a secondcoordinate based on the first coordinate.

In some embodiments, the second obtaining module 120 is specificallyconfigured to obtain a first 2D coordinate of the support key point ofthe second-type part based on a 2D image; and obtain a first 3Dcoordinate corresponding to the first 2D coordinate based on the first2D coordinate and a conversion relationship between a 2D coordinate anda 3D coordinate.

In some embodiments, the second obtaining module 120 is specificallyconfigured to obtain a second 3D coordinate of the support key point ofthe second-type part of the target based on a 3D image; and obtain athird 3D coordinate based on the second 3D coordinate.

In some embodiments, the second obtaining module 120 is specificallyconfigured to correct, based on the second 3D coordinate, a 3Dcoordinate of a support key point corresponding to an occluded portionof the second-type part in the 3D image so as to obtain the third 3Dcoordinate.

In some embodiments, the first determining module 130 is specificallyconfigured to determine a quaternion of the second-type part based onthe position information.

In some embodiments, the second obtaining module 120 is specificallyconfigured to: obtain first position information of the support keypoint of a first part in the second-type part; and obtain secondposition information of the support key point of a second part in thesecond-type part.

In some embodiments, the first determining module 130 is specificallyconfigured to: determine movement information of the first partaccording to the first position information; and determine movementinformation of the second part according to the second positioninformation.

In some embodiments, the control module 140 is specifically configuredto: control movement of a part in the controlled model corresponding tothe first part according to the movement information of the first part;and control movement of a part in the controlled model corresponding tothe second part according to the movement information of the secondpart.

In some embodiments, the first part is a torso; and the second part isupper limbs, lower limbs, or four limbs.

Several specific examples are provided below with reference to any oneof the foregoing embodiments.

Example 1

This example provides an image processing method, including thefollowing steps.

An image is captured, where the image includes a target, and the targetincludes, but is not limited to, a human body.

Face key points of the human body are detected, where the face keypoints may be contour key points of a face surface.

Torso key points and/or limb key points of the human body are detected,where the torso key points and/or the limb key points herein may all be3D key points and are represented by 3D coordinates. The 3D coordinatesmay be 3D coordinates obtained by detecting 2D coordinates from a 2Dimage and then using a conversion algorithm from the 2D coordinates tothe 3D coordinates. The 3D coordinates may also be 3D coordinatesextracted from a 3D image captured by a 3D camera. The limb key pointsherein may include upper limb key points and/or lower limb key points.Taking the hand as an example, hand key points of the upper limb keypoints include, but are not limited to, wrist joint key points,metacarpophalangeal joint key points, knuckle joint key points, andfingertip key points. The positions of these key points may reflectmovements of the hand and fingers.

Mesh information of the face is generated according to the face keypoints. An expression base corresponding to the current expression ofthe target is selected according to the mesh information, and theexpression of the controlled model is controlled according to theexpression base; and the expression strength of the controlled modelcorresponding to each expression base is controlled according to anintensity coefficient reflected by the mesh information.

Quaternions are converted according to the torso key points and/or thelimb key points. The torso movement of the controlled model iscontrolled according to quaternions corresponding to the torso keypoints; and/or the limb movement of the controlled model is controlledaccording to quaternions corresponding to the limb key points.

For example, the face key points may include 106 key points. The torsokey points and/or the limb key points may include 14 key points or 17key points, specifically as shown in FIG. 7A and FIG. 7B. FIG. 7A is aschematic diagram including 14 skeleton key points. FIG. 7B is aschematic diagram including 17 skeleton key points.

FIG. 7B may be a schematic diagram including 17 key points generatedbased on the 14 key points shown in FIG. 7A. The 17 key points in FIG.7B are equivalent to the key points shown in FIG. 7A with the additionof key point 0, key point 7, and key point 9. The 2D coordinates of keypoint 9 may be preliminarily determined based on the 2D coordinates ofkey point 8 and key point 10, and the 2D coordinates of key point 7 maybe determined according to the 2D coordinates of key point 8 and the 2Dcoordinates of key point 0. Key point 0 may be a reference pointprovided by the embodiments of the present disclosure, and the referencepoint may serve as the foregoing first reference point and/or secondreference point.

In this example, the controlled model may be a game character in a gamescenario, a teacher model in an online education video in an onlineteaching scenario, or a virtual anchor in a virtual webcasting scenario.In short, the controlled model is determined according to theapplication scenario. If the application scenario is different, themodel and/or appearance of the controlled model is different.

For example, in a conventional platform teaching scenario, such asmathematics, physics, or the like, the clothes of the teacher model maybe relatively formal, such as a suit. For another example, for a sportsteaching scenario such as yoga or gymnastics, the clothes of thecontrolled model may be sportswear.

Example 2

This example provides an image processing method, including thefollowing steps.

An image is captured, where the image includes a target, and the targetincludes, but is not limited to, a human body.

Torso key points and/or limb key points of the human body are detected,where the torso key points and/or the limb key points here may all be 3Dkey points and are represented by 3D coordinates. The 3D coordinates maybe 3D coordinates obtained by detecting 2D coordinates from a 2D imageand then using a conversion algorithm from the 2D coordinates to the 3Dcoordinates. The 3D coordinates may also be 3D coordinates extractedfrom a 3D image captured by a 3D camera. The limb key points herein mayinclude upper limb key points and/or lower limb key points. Taking thehand as an example, hand key points of the upper limb key pointsinclude, but are not limited to, wrist joint key points,metacarpophalangeal joint key points, knuckle joint key points, andfingertip key points. The positions of these key points may reflectmovements of the hand and fingers.

The torso key points are converted into quaternions that represent atorso movement. The quaternions may be referred to as torso quaternions.The limb key points are converted into quaternions that represent a limbmovement. The quaternions may be referred to as limb quaternions.

The torso movement of the controlled model is controlled by the torsoquaternions. The limb movement of the controlled model is controlled bythe limb quaternions.

The torso key points and/or the limb key points may include 14 keypoints or 17 key points, specifically as shown in FIG. 7A or FIG. 7B.

In this example, the controlled model may be a game character in a gamescenario, a teacher model in an online education video in an onlineteaching scenario, or a virtual anchor in a virtual webcasting scenario.In short, the controlled model is determined according to theapplication scenario. If the application scenario is different, themodel and/or appearance of the controlled model is different.

For example, in a conventional platform teaching scenario, such asmathematics, physics, or the like, the clothes of the teacher model maybe relatively formal, such as a suit. For another example, for a sportsteaching scenario such as yoga or gymnastics, the clothes of thecontrolled model may be sportswear.

Example 3

This example provides an image processing method, including thefollowing steps.

An image is obtained, where the image includes a target, and the targetmay be a human body.

A 3D posture of the target in a 3D space is obtained according to theimage, where the 3D posture may be represented by 3D coordinates ofskeleton key points of the human body.

An absolute rotation parameter of a joint of the human body in a cameracoordinate system is obtained, where the absolute rotation position maybe determined by coordinates in the camera coordinate system.

A coordinate direction of the joint is obtained according to thecoordinates. A relative rotation parameter of the joint is determinedaccording to a hierarchical relationship. Determining the relativeparameter may specifically include: determining the position of a keypoint of the joint with respect to a root node of the human body. Therelative rotation parameter may be represented by a quaternion. Thehierarchical relationship herein may be a pull relationship betweenjoints. For example, the movement of an elbow joint may drive themovement of a wrist joint to some extent, the movement of a shoulderjoint may also drive the movement of the elbow joint, or the like. Thehierarchical relationship may also be predetermined according to jointsof the human body.

The rotation of the controlled model is controlled by the quaternion.

For example, the following is an example of one hierarchicalrelationship. The first level: pelvis; the second level: waist; thethird level: thighs (e.g., left thigh and right thigh); the fourthlevel: caves (e.g., left crus and right crus); and the fifth level:feet.

For another example, the following is another hierarchical relationship.The first level: chest; the second level: neck; and the third level:head.

Further, for example, the following is still another hierarchicalrelationship. The first level: clavicles, corresponding to the shoulder;the second level: upper arms; the third level: forearms (also referredto as lower arms); and the fourth level: hand.

From the first level to the fifth level, the hierarchical relationshipdecreases in sequence. The movement of the part at the upper levelaffects the movement of the part at the lower level. Therefore, thelevel of the pull portion is higher than that of the connecting portion.

During determination of the second type of movement information, first,the movement information of the key points of the part at each level isobtained, and then based on the hierarchical relationship, the movementinformation (i.e., the relative rotation information) of the key pointsof the part at the low level with respect to the key points of the partat the high level is determined.

For example, if a quaternion is used for representing movementinformation, the relative rotation information may be represented by thefollowing calculation formula. A rotation quaternion of each key pointwith respect to the camera coordinate system is {Q₀, Q₁, . . . , Q₁₈},and then a rotation quaternion q_(i) of each key point with respect to aparent key point is calculated according to formula (1).

Controlling the movement of each joint of the controlled model by usingthe quaternion may include: controlling the movement of each joint ofthe controlled model using q_(i).

A further image processing method further includes: converting thequaternion into a first Euler angle; transforming the first Euler angleto obtain a second Euler angle within a constraint condition, where theconstraint condition may be used for performing angle limitation on thefirst Euler angle; and obtaining a quaternion corresponding to thesecond Euler angle, and then controlling the rotation of the controlledmodel by using the quaternion. Obtaining the quaternion corresponding tothe second Euler angle may be: directly converting the second Eulerangle into a quaternion.

Taking the human body as an example, 17 joint key points may be detectedby means of human body detection. In addition, two key points are alsoset corresponding to left and right hands. Therefore, there are a totalof 19 key points. FIG. 7B is a schematic diagram of the skeletonincluding 17 key points. FIG. 8 is a schematic diagram of the skeletonincluding 19 key points. The bones shown in FIG. 8 may correspond to 19key points, which are pelvis, waist, left thigh, left crus, left foot,right thigh, right crus, right foot, chest, neck, head, left clavicle,right clavicle, right upper arm, right forearm, right hand, left upperarm, left forearm, left hand, respectively.

First, coordinates of the 17 key points in an image coordinate systemcan be obtained by detecting key points of joints of the human body inan image, specifically as follows: S={(x₀, y₀, z₀), . . . , (x₁₆, y₁₆,z₁₆)} (x_(i),y_(i),z_(i)) may be coordinates of the 1^(−th) key point,and the value of i is from 0 to 16.

Coordinates of the 19 joint key points in respective local coordinatesystems may be defined as follows: A={(p₀, q₀), . . . , (p₁₈, q₁₈)},where p_(i) represents 3D coordinates of node i in a local coordinatesystem, is generally a fixed value carried by an original model, anddoes not need to be modified or migrated. q_(i) is a quaternion,represents the rotation of a bone controlled by node i in a coordinatesystem of its parent node, and may also be regarded as the rotation ofthe local coordinate system of the current node and the local coordinatesystem of the parent node.

The process for calculating quaternions of key points corresponding tothe joints is as follows: determining coordinate axis directions of alocal coordinate system of each node. For each bone, a direction inwhich a child node points to a parent node is an x-axis; a rotation axisthat maximizes the rotation angle of the bone is taken as a z-axis; andif the rotation axis cannot be determined, a direction facing the humanbody is taken as a y-axis. Specifically, reference is made to FIG. 9.

In this example, a left-hand coordinate system is used for explaining,and a right-hand coordinate system may also be used in specificimplementation.

Serial numbers of nodes in a Calculation with key points 19-nodeskeleton in a 17-point skeleton 0 Take (1-7) × (1-4) as a y-axis and(7-0) as an x-axis 1 Take maximum default values of 3D coordinates 2Take (14-11) × (14-7) as a y-axis and (8-7) as an x-axis 3 Take maximumdefault values of 3D coordinates 4 Take (10-8) as an x-axis and (9-10) ×(9-8) as a z-axis 5 Take (11-8) as an x-axis and (12-11) × (11-8) as ay-axis 6 Take (12-11) as an x-axis and (11-12) × (12-13) as a z-axis 7Take (13-12) as an x-axis and (11-12) × (12-13) as a z-axis. Note: thenode changes after a quaternion of the hand is added subsequently. 9Take (5-4) as an x-axis and (5-6) × (5-4) as a z-axis 10 Take (6-5) asan x-axis and (5-6) × (5-4) as a z-axis 12 Take (14-8) as an x-axis and(8-14) × (14-15) as a y-axis 13 Take (15-14) as an x-axis and (14-15) ×(15-16) as a z-axis 14 Take (16-15) as an x-axis and (14-15) × (15-16)as a z-axis. Note: the node changes after a quaternion of the hand isadded subsequently. 16 Take (2-1) as an x-axis and (2-3) × (2-1) as az-axis 17 Take (3-2) as an x-axis and (2-3) × (2-1) as a z-axis

(i-j) in the table above represents a vector where i points to j, and xpresents cross-multiplying. For example, (1-7) represents a vector wherethe first key point points to the seventh key point.

In the table above, nodes 8, 15, 11, and 18 are four nodes of the handsand feet. Calculation of quaternions of the four nodes may be determinedonly when specific postures are used, and therefore, the four nodes arenot included in this table. In addition, in the table above, the serialnumbers of the nodes of the 19-node skeleton may be seen in FIG. 8, andthe serial numbers of the nodes of the 17-node skeleton may be seen inFIG. 7B.

The process for calculating the first Euler angle is as follows.

After a local rotation quaternion q_(i) of joint points is calculated,and it is converted into an Euler angle first, where the order of x-y-zis used by default.

q_(i)=(q0,q1,q2,q3) is set, where q0 is a real number; and q1, q2, andq3 are all imaginary numbers. Therefore, the calculation formulas(2)-(4) of the Euler angle are as follows:

X=a tan 2(2*(q0q1−q2*q3),1−2*(q1*q1+q2*q2))   (2)

Y=a sin(2*(q1*q3+q0*q2)) and the value of Y ranges from −1 to 1  (3)

Z=a tan 2(2*(q0*q3−q1*q2),1−2*(q2*q2+q3*q3))  (4)

X is an Euler angle in a first direction; Y is an Euler angle in asecond direction; and Z is an Euler angle in a third direction. Any twoof the first direction, the second direction, and the third directionare perpendicular to each other.

Then, the three angles (X,Y,Z) may be limited, if exceeding the range,the angles are limited to boundary values, and the second Euler angle(X′,Y′,Z′) after correction is obtained. A new local coordinate systemrotation quaternion q_(i)′ is formed by restoration.

Another further image processing method further includes: performingposture optimization adjustment on the second Euler angle. For example,some angles of the second Euler angle are adjusted, and the second Eulerangle is adjusted into a posture-optimized Euler angle based on a presetrule so as to obtain a third Euler angle. Therefore, obtaining thequaternion corresponding to the second Euler angle includes: convertingthe third Euler angle into a quaternion controlling the controlledmodel.

Still another further image processing method further includes: afterconverting the second Euler angle into a quaternion, performing postureoptimization processing on quaternion obtained after the conversion. Forexample, adjustment is performed based on a preset rule to obtain anadjusted quaternion and the controlled model is controlled according tothe finally adjusted quaternion.

In some embodiments, the second Euler angle or the quaternion obtainedby conversion of the second Euler angle may be adjusted based on apreset rule, or may be automatically optimized and adjusted by a deeplearning model. There are a variety of specific implementations, and nolimitation is made in the present application.

In addition, still another image processing method further includespre-processing. For example, according to the size of the captured humanbody, the width of the hip and/or the shoulder of the controlled modelare modified to correct the overall posture of the human body. Uprightstanding correction and lifting abdomen correction may be performed onthe standing posture of the human body. Some persons may lift theirstomachs when standing, and the stomach lifting correction enables thecontrolled model not to simulate the user's stomach lifting action. Somepersons may stoop when standing, and stooping correction enables thecontrolled model not to simulate the user's stooping action or the like.

Example 4

This example provides an image processing method, including thefollowing steps.

An image is obtained, where the image includes a target, and the targetmay include at least one of a human body, a human upper limb, or a humanlower limb.

A coordinate system of a target joint is obtained according to positioninformation of the target joint in an image coordinate system. Acoordinate system of the limb part that may drive the target joint tomove is obtained according to position information of a limb part in theimage coordinate system.

Rotation of the target joint with respect to the limb part is determinedbased on the coordinate system of the target joint and the coordinatesystem of the limb part, to obtain a rotation parameter, where therotation parameter includes a self-rotation parameter of the targetjoint and a parameter of rotation driven by the limb part.

The parameter of rotation driven by the limb part is limited by a firstangle limit to obtain a final driven-rotation parameter. The rotationparameter of the limb part is corrected according to the finaldriven-rotation information. A relative rotation parameter is obtainedaccording to the coordinate system of the limb part and the correctedrotation parameter of the limb part. Second angle limitation isperformed on the relative rotation information to obtain a limitedrelative rotation parameter.

A quaternion is obtained from the limited rotation information. Themovement of the target joint of the controlled model is controlledaccording to the quaternion.

For example, if the human upper limb is processed, a coordinate systemof the hand in the image coordinate system is obtained, and a coordinatesystem of a lower arm and a coordinate system of an upper arm areobtained. In this case, the target joint is a wrist joint. Rotation ofthe hand with respect to the lower arm is split into self-rotation anddriven rotation. The driven rotation is transferred to the lower arm,specifically, for example, the driven rotation is assigned to rotationof the lower arm in a corresponding direction; and the maximum rotationof the lower arm is limited by first angle limitation of the lower arm.Then, the rotation of the hand with respect to the corrected lower armis determined to obtain a relative rotation parameter. Second anglelimitation is performed on the relative rotation parameter to obtain therotation of the hand with respect to the lower arm.

If a human lower limb is processed, a coordinate system of the foot inthe image coordinate system is obtained, and a coordinate system of thecrus and a coordinate system of the thigh are obtained. In this case,the target joint is the ankle joint. Rotation of the foot with respectto the crus is split into self-rotation and driven rotation. The drivenrotation is transferred to the crus, specifically, for example, thedriven rotation is assigned to rotation of the crus in a correspondingdirection; and the maximum rotation of the crus is limited by firstangle limitation of the crus. Then, the rotation of the foot withrespect to the corrected crus is determined to obtain a relativerotation parameter. Second angle limitation is performed on the relativerotation parameter to obtain the rotation of the foot with respect tothe crus.

Example 5

The neck controls the orientation of the head, and the face, body, andhands are separate components. Ultimately, the rotation of the neck isvery important to form a whole.

An orientation of the human body may be calculated according to keypoints of the human body. According to key points of the face, anorientation of the face may be calculated. The relative position ofthese two orientations is a rotation angle of the neck. The problem ofthe angle of a connecting portion is to be solved. Such problem of theangle of the connecting portion is solved by means of relativecalculation. For example, if the body is at 0 degree and the face is at90 degrees, to control a controlled model, it only pays attention to alocal angle. For the change in the angles of the head and the body, andthe angle of the neck of the controlled model needs to be calculated tocontrol the head of the controlled model.

In this example, first, the orientation of the user's current face isdetermined based on the image, and then the rotation angle of the neckis calculated. Because the rotation of the neck is within a range, forexample, it is assumed that the neck may rotate 90 degrees at most. Ifthe calculated rotation angle exceeds this range (−90 degrees to 90degrees), the boundary of the range is taken as the rotation angle ofthe neck (e.g., −90 degrees or 90 degrees).

3D key points may be used to calculate the orientation of the body orface. The calculation of the specific orientation may be: two vectors ina plane where the face or body is located that are not in a straightline are cross-multiplied to obtain a normal vector of the plane, andthe normal vector is the orientation of the face or body. Thisorientation may be taken as the orientation of the connecting portion(neck) between the body and the face.

As shown in FIG. 10, the embodiments of the present application providean image device, including: a memory 1002, configured to storeinformation; and a processor 1001, connected to the memory 1002 andconfigured to execute computer-executable instructions stored on thememory 1002 so as to implement the image processing method provided byone or more of the foregoing technical solutions, for example, the imageprocessing method shown in FIG. 1 and/or FIG. 2.

The memory 1002 may be various types of memories, such as a randomaccess memory, a Read-Only Memory (ROM), and a flash memory. The memory1002 may be used for information storage, for example, storingcomputer-executable instructions or the like. The computer-executableinstructions may be various program instructions, such as target programinstructions and/or source program instructions.

The processor 1001 may be various types of processors, such as a centralprocessing unit, a microprocessor, a digital signal processor, aprogrammable array, a digital signal processor, an application-specificintegrated circuit, or a graphic processing unit.

The processor 1001 may be connected to the memory 1002 by means of abus. The bus may be an integrated circuit bus or the like.

In some embodiments, the terminal device may further include: acommunication interface 1003. The communication interface 1003 mayinclude: a network interface such as a local area network interface anda transceiver antenna. The communication interface is also connected tothe processor 1001 and may be used for information transmission andreception.

In some embodiments, the terminal device further includes ahuman-computer interaction interface 1005. For example, thehuman-computer interaction interface 1005 may include variousinput/output devices such as a keyboard and a touch screen.

In some embodiments, the image device further includes: a display 1004which may display various prompts, captured face images, and/or variousinterfaces.

The embodiments of the present application provide a non-volatilecomputer storage medium, having a computer-executable code storedthereon, where after the computer-executable code is executed, the imageprocessing method provided by one or more of the foregoing technicalsolutions, for example, the image processing method as shown in FIG. 1and/or FIG. 2, is implemented.

It should be understood that the device and method disclosed in severalembodiments provided in the present application may be implemented inother manners. The device embodiments described above are merelyexemplary. For example, the unit division is merely logical functiondivision and may be actually implemented in other division manners. Forexample, a plurality of units or components may be combined orintegrated into another system, or some features may be ignored or notperformed. In addition, the displayed or discussed mutual couplings ordirect couplings or communication connections among the components maybe implemented by means of some interfaces. The indirect couplings orcommunication connections between the devices or units may beelectrical, mechanical, or in other forms.

The units described as separate components may or may not be physicallyseparate, and the components displayed as units may or may not bephysical units, i.e., may be located at one position, or may bedistributed on a plurality of network units. Some or all of the unitsmay be selected according to actual needs to achieve the objectives ofthe solutions of the embodiments.

In addition, the functional units in the embodiments of the presentdisclosure may be all integrated into one processing module, or each ofthe units may respectively serve as an independent unit, or two or moreunits are integrated into one unit, and the integrated unit may beimplemented in the form of hardware, or may also be implemented in theform of a hardware and software function unit.

A person of ordinary skill in the art may understand that all or somesteps for implementing the foregoing method embodiments may be completedby a program by instructing related hardware; the foregoing program maybe stored in a computer-readable storage medium; when the program isexecuted, steps including the foregoing method embodiments areperformed. Moreover, the foregoing non-volatile storage medium includesvarious media capable of storing a program code, such as a mobilestorage device, an ROM, a magnetic disk, or an optical disk.

The descriptions above are only specific implementations of the presentdisclosure. However, the scope of protection of the present disclosureis not limited thereto. Within the technical scope disclosed by thepresent disclosure, any variation or substitution that can be easilyconceived of by those skilled in the art should all fall within thescope of protection of the present disclosure. Therefore, the scope ofprotection of the present disclosure should be determined by the scopeof protection of the claims.

1. An image processing method, comprising: obtaining an image; obtaininga feature of a part of a target based on the image; determining movementinformation of the part based on the feature; and controlling movementof a corresponding part in a controlled model according to the movementinformation.
 2. The method according to claim 1, wherein obtaining thefeature of the part of the target based on the image comprises:obtaining a first-type feature of a first-type part of the target basedon the image; and/or obtaining a second-type feature of a second-typepart of the target based on the image.
 3. The method according to claim2, wherein obtaining the first-type feature of the first-type part ofthe target based on the image comprises: obtaining an expression featureof a head and an intensity coefficient of the expression feature basedon the image.
 4. The method according to claim 3, wherein obtaining theintensity coefficient of the expression feature based on the imagecomprises: obtaining, based on the image, an intensity coefficient thatrepresents each sub-part in the first-type part.
 5. The method accordingto claim 3, wherein determining the movement information of the partbased on the feature comprises: determining movement information of thehead based on the expression feature and the intensity coefficient; andcontrolling the movement of the corresponding part in the controlledmodel according to the movement information comprises: controlling anexpression change of a head in the controlled model according to themovement information of the head.
 6. The method according to claim 2,wherein obtaining the second-type feature of the second-type part of thetarget based on the image comprises: obtaining position information of akey point of the second-type part of the target based on the image; anddetermining the movement information of the part based on the featurecomprises: determining movement information of the second-type partbased on the position information.
 7. The method according to claim 6,wherein obtaining the position information of the key point of thesecond-type part of the target based on the image comprises: obtaining afirst coordinate of a support key point of the second-type part of thetarget based on the image; and obtaining a second coordinate based onthe first coordinate.
 8. The method according to claim 7, whereinobtaining the first coordinate of the support key point of thesecond-type part of the target based on the image comprises: obtaining afirst 2-Dimensional (2D) coordinate of the support key point of thesecond-type part based on a 2D image; and obtaining the secondcoordinate based on the first coordinate comprises: obtaining a first3-Dimensional (3D) coordinate corresponding to the first 2D coordinatebased on the first 2D coordinate and a conversion relationship between a2D coordinate and a 3D coordinate.
 9. The method according to claim 7,wherein obtaining the first coordinate of the support key point of thesecond-type part of the target based on the image comprises: obtaining asecond 3D coordinate of the support key point of the second-type part ofthe target based on a 3D image; and obtaining the second coordinatebased on the first coordinate comprises: obtaining a third 3D coordinatebased on the second 3D coordinate.
 10. The method according to claim 9,wherein obtaining the third 3D coordinate based on the second 3Dcoordinate comprises: correcting, based on the second 3D coordinate, a3D coordinate of a support key point corresponding to an occludedportion of the second-type part in the 3D image, to obtain the third 3Dcoordinate.
 11. The method according to claim 6, wherein determining themovement information of the second-type part based on the positioninformation comprises: determining a quaternion of the second-type partbased on the position information.
 12. The method according to claim 6,wherein obtaining the position information of the key point of thesecond-type part of the target based on the image comprises: obtainingfirst position information of a support key point of a first part in thesecond-type part; and obtaining second position information of a supportkey point of a second part in the second-type part.
 13. The methodaccording to claim 12, wherein determining the movement information ofthe second-type part based on the position information comprises:determining movement information of the first part according to thefirst position information; and determining movement information of thesecond part according to the second position information.
 14. The methodaccording to claim 13, wherein controlling the movement of thecorresponding part in the controlled model according to the movementinformation comprises: controlling movement of a part in the controlledmodel corresponding to the first part according to the movementinformation of the first part; and controlling movement of a part in thecontrolled model corresponding to the second part according to themovement information of the second part.
 15. The method according toclaim 12, wherein the first part is a torso; and/or the second part isan upper limb, a lower limb, or four limbs.
 16. An image device,comprising: a memory storing computer-executable instructions; and aprocessor coupled to the memory, wherein the processor is configured toobtain an image; obtain a feature of a part of a target based on theimage; determine movement information of the part based on the feature;and control movement of a corresponding part in a controlled modelaccording to the movement information.
 17. The device according to claim16, wherein obtaining the feature of the part of the target based on theimage comprises: obtaining a first-type feature of a first-type part ofthe target based on the image; and/or obtaining a second-type feature ofa second-type part of the target based on the image.
 18. The deviceaccording to claim 17, wherein obtaining the first-type feature of thefirst-type part of the target based on the image comprises: obtaining anexpression feature of a head and an intensity coefficient of theexpression feature based on the image.
 19. The device according to claim17, wherein obtaining the second-type feature of the second-type part ofthe target based on the image comprises: obtaining position informationof a key point of the second-type part of the target based on the image;and determining the movement information of the part based on thefeature comprises: determining movement information of the second-typepart based on the position information.
 20. A non-volatile computerstorage medium storing computer-executable instructions that areexecuted by a processor to: obtain an image; obtain a feature of a partof a target based on the image; determine movement information of thepart based on the feature; and control movement of a corresponding partin a controlled model according to the movement information.